Using the SAS System to Examine the Agreement between Two Programs That Score Surveys Using Samejima’s Graded Response Model

In the Spring of 1998, researchers at the Center for Creative Leadership chose to use the Graded Response Model by Samejima as part of a project to assess the behavioral changes that are seen in the participants in the leadership development programs of the Center. It was determined that the existing DOS and Windows software would not function with sufficient reliability within the extant computing network of the Center, and a contract was awarded to a vendor for the development of scoring program that would operate reliably at the Center. As a part of the contract specifications, the Item Response Theory parameter estimates of ability and standard errors that were produced by the new computing module had to agree with those estimates produced by the existing software. As one might expect, it was difficult to demonstrate exact convergence of the two sets of estimates. This paper presents the statistical process by which the case for sufficient convergence was made.